fix: preserve sandbox recovery readiness guard - #7291

<!-- markdownlint-disable MD041 -->
## Summary
`nemoclaw <sandbox> stop` returned success but never tore down the
host-side dashboard SSH port-forward it created, leaving an `ssh -L
127.0.0.1:<port>` listener alive. `status` then misreported the
cleanly-stopped sandbox as a foreign `sandbox_dashboard_port_conflict`,
and `start`/`recover` intermittently failed while contending for the
still-held port. This PR releases the forward on stop. Exact-head live
`double-onboard` also exposed the known recreated-sandbox readiness
defect from #7273, so this branch includes the reviewed tri-state
readiness fix from #7291 and keeps the restart regression enabled.

Closes #7227.
Closes #7273.

## Reproduction
On our Ubuntu 24.04 x86_64 test host (no GPU), ollama-local sandbox. The
reporter is on DGX Spark aarch64; the root cause (host-side port-forward
not torn down on stop) is platform-independent.

1. Onboard an OpenClaw sandbox on ollama-local; confirm `Phase: Ready`.
2. `openshell forward list` / `ss -ltnp | grep <dashboard-port>` — a
nemoclaw-spawned `ssh -L` forward holds the port.
3. `nemoclaw <sb> stop` (exit 0; container Exited 137).
4. Re-check the port and `nemoclaw <sb> status`.

**Observed on `main` (before fix)**
```
# after stop, the forward is still alive:
$ ss -ltnp | grep 18789
LISTEN 127.0.0.1:18789 ... users:(("ssh",pid=<same as before stop>,fd=3))
$ openshell forward list
<sb> 127.0.0.1 18789 <pid> running
$ nemoclaw <sb> status            # EXIT 1
Failure layer: sandbox_dashboard_port_conflict — sandbox container is stopped and the dashboard port is held by a foreign listener.
```

**Observed on `fix/...` (after fix)**
```
# after stop, no listener remains:
$ ss -ltnp | grep 18789            # 0 listeners
$ openshell forward list           # 0 entries for <sb>
$ nemoclaw <sb> status
Failure layer: sandbox_container_stopped — sandbox container exists but is not running.
# repeated start/stop cycles (x3): start always reaches Ready; stop always leaves 0 listeners.
```

## Analysis
`stopSandbox` (`src/lib/actions/sandbox/stop.ts`) stops the labeled
container(s) and returns, but it never releases the host-side dashboard
port-forward. That forward is a nemoclaw-managed `openshell forward`
(`ssh -L 127.0.0.1:<port>`); onboarding, destroy, and forward-recovery
all tear it down with `openshell forward stop <port> <sandbox>`, but
`stop` did not — so the listener outlives the container.
`getSandboxGatewayStateForStatus`'s classifier
(`gateway-failure-classifier.ts`, `sandbox_dashboard_port_conflict`)
then sees a stopped container plus a live listener on its dashboard port
and reports a "foreign listener" conflict — the listener is nemoclaw's
own leftover forward. `start`/`recover` re-establish the forward and
probe gateway readiness while the just-stopped forward is still bound,
which is the intermittent failure.

The exact-head E2E gate also exposed a finalization ordering gap:
`cloud-onboard` created the sandbox and launched OpenClaw, but
post-policy process recovery invalidated the initial dashboard forward
before final deployment verification. Finalization re-established
forwards only for non-default agents and did so before recovery, so the
delivered OpenClaw dashboard on `127.0.0.1:18789` returned connection
refused. Existing unit review did not model recovery invalidating the
forward; the new regression makes that state transition explicit.

## Fix
After the container is stopped, release the forward via a new
`teardownSandboxDashboardForward` helper (`forward-recovery.ts`) that
runs `openshell forward stop <port> <sandbox>` — reusing the exact
teardown already used by onboard/destroy/forward-recovery, with the port
from `resolveSandboxDashboardPort`. The helper scopes cleanup to the
registry-owned gateway, applies the standard 30-second OpenShell
operation timeout, and waits up to 5 seconds for the supported OpenShell
0.0.85 listener to release the host port. It is best-effort (a stop must
still free container resources when OpenShell is unreachable) and is
wired into `stopSandbox` after a successful Docker stop and for an
already-stopped sandbox. It does not run when Docker stop fails because
the container is still running and needs its forward.

Tests (`stop.test.ts`) pin the contract: the forward is torn down only
after the container stops; it is not released when Docker stop fails;
and a repeated stop releases a leftover forward for an already-at-rest
sandbox. They also verify non-default gateway targeting, the operation
timeout, delayed listener-release polling, and that a nonzero cleanup
status skips the port probe.

Finalization now reconciles the default OpenClaw or declared-agent
forward after post-policy process recovery and before warm-up or
deployment verification. The regression simulates recovery invalidating
an initially-live forward and requires finalization to restore it before
reporting the deployment healthy.

## Changes
- `src/lib/actions/sandbox/process-recovery.ts` and focused tests:
preserve the reviewed #7291 tri-state readiness contract and retry only
the exact OpenShell supervisor re-registration states and no-op
readiness-probe timeout exposed by exact-head `double-onboard`.
- `test/e2e/live/double-onboard.test.ts`: retain the stop/start
regression so live validation proves the recovery dependency instead of
bypassing it.
- `src/lib/actions/sandbox/forward-recovery.ts`: new
`teardownSandboxDashboardForward` helper.
- `src/lib/actions/sandbox/stop.ts`: release the dashboard forward after
a successful stop.
- `src/lib/actions/sandbox/stop.test.ts`: pin the teardown contract and
its boundaries.
- `src/lib/onboard.ts` and finalization tests: restore dashboard
forwarding after post-policy process recovery, including the default
OpenClaw path.
- `src/lib/onboard/policy-selection.ts` and focused tests: after
applying policy presets, require both sandbox phase readiness and a real
OpenShell command probe before finalization.
- Timeout documentation: map post-policy OpenShell re-registration to
`NEMOCLAW_SANDBOX_READY_TIMEOUT` and distinguish it from post-create
cleanup.
- `docs/manage-sandboxes/run-sandboxes.mdx`: document dashboard-forward
cleanup during stop.
- `docs/reference/commands.mdx`: document cleanup ordering, failure
handling, and idempotency.

## Type of Change
- [x] Code change (feature, bug fix, or refactor)

## Verification
- [x] Focused recovery tests pass: 32/32.
- [x] Focused finalization tests pass: 16/16.
- [x] Exact affected Vitest suite against current `main` passes: 107
files / 1,060 tests.
- [x] CLI type-check, CLI build, Biome, semantic E2E plans, `git diff
--check`, normal pre-commit/commit-msg hooks, pre-push hooks, and
gitleaks pass.
- [x] `npm run docs` passes with 0 errors and 2 existing warnings; agent
variants are synchronized and valid.
- [ ] Replacement exact-head live E2E is pending for the OpenShell
relay-readiness and final-forward ordering fixes.
- [x] No secrets, API keys, or credentials committed

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: PASS on exact head
`f7acc8049d1cccb914cbb7328bbfcd4deb378814` against base
`04e6dfa883071dda9df429c66e73168e1a995cba`. The reviewer inspected the
full 22-file PR diff, including both exact OpenShell 0.0.85
relay-readiness timeout forms, refreshed-main resolution, final
recovery/forward reconciliation, and all agent variants. Existing
documentation sufficiently covers the bounded readiness, fail-closed
recovery, forward cleanup, and final verification behavior. `npm run
docs`, agent-variant validation, and `git diff --check` passed.
- Agent: `Codex Desktop`
<!-- docs-review-head-sha: f7acc80 -->
<!-- docs-review-agents-blob-sha: 9c9b36d -->

## AI Disclosure
- [x] AI-assisted — tool: Claude Code

Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved sandbox stop to best-effort release the host dashboard
port-forward, including idempotent cleanup and safer degradation when
OpenShell isn’t available.
* Strengthened recovery readiness handling with bounded, per-stage
timeouts, clearer retry vs fail-closed outcomes, and better gating
before starting port-forwards.
* Updated onboarding finalization to restore the correct dashboard
forward after recovery and to gate preset synchronization on
control-plane readiness.
* **Tests**
* Expanded unit, supervisor relaunch, onboarding, and e2e coverage for
forward release, readiness retries, and edge-case recovery flows.
* **Documentation**
* Refreshed stop/recovery and timeout guidance to match the preserved
assets and the updated dashboard forward behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
Signed-off-by: ScarabSystems <scarab.systems@yahoo.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: ScarabSystems <scarab.systems@yahoo.com>
Co-authored-by: Charan Jagwani <cjagwani@nvidia.com>
Co-authored-by: Prekshi Vyas <34834085+prekshivyas@users.noreply.github.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>

Record NVIDIA/NemoClaw#7406 as an accepted #7387 repair after upstream merge.

Refresh NVIDIA/NemoClaw#7254 as accepted test coverage for #6042 while noting the issue remains open and the merged PR was test-only.

Refresh NVIDIA/NemoClaw#7291/#7273 as closed without merge so the Field Lab no longer reports that PR as open.

Verification: gh pr/issue views for #7254, #7406, #7291, #6042, #7387, and #7273; git diff --cached --check; public leakage scan for local/private/SDS terms.